JsPsychR

Open source, standard tooling for experimental protocols: towards Registered reports

Gorka Navarrete & Herman Valencia

The past

Running experiments


Old school:

  1. Read a bit & come up with an idea
  2. Prepare and run experiment
  3. Prepare data & explore different analytic approaches
  4. Significant result → Write paper

Some issues

https://www.cos.io/initiatives/registered-reports

Experimenter degrees of freedom, incentives, issues

  • What if… → Garden of forking paths (Rubin 2017)

  • p-hacking, hypothesizing after the results are known (i.e. HARKing)

  • The need for significance and novelty

  • False positives research

Psychology Replication crisis

  • Replication of 100 studies
  • Replication effects were half the magnitude of original effects
  • p < 0.05:
    • 97% of original studies
    • 36% of replications

Open-Science-Collaboration (2015)

Is there a crisis? Why?

Baker (2016)

Improving Replicability

  • Improve methods: increase n, better measures and manipulations, improve design, piloting

  • Reduce temptations: p-hacking, hypothesizing after the results are known (i.e. HARKing), selective reporting → preregistration, internal replications

  • Openness: transparency of the research process, sharing methods, materials, procedures, and data

Nosek et al. (2022)

Registered reports (RR)



RRs were conceived to alter the incentives for authors and journals away from producing novel, positive, clean findings and towards conducting and publishing rigorous research on important questions. Soderberg et al. (2021)

RR do help

How RR work

  • Write introduction, method, … before collecting data!
  • Send to journal for review
  • Revise and resubmit (improve before collecting data)
  • Once you get In principle acceptance, collect human data, run analysis, write up, and send for a final review

RR advantages

  • More open, preregistered, reproducible by default

  • It does not matter if p value is < 0.05

  • Less incentives for p-hacking

  • More trustworthy results

  • You still can explore, but have to say explicitly

Registered reports are great

But isn’t this a bit…



  • Before having the data available, it is hard to know how to analyze it

  • There are always surprises when receiving new data. How can I create an analysis plan that will hold?

Our path towards RR



Background

We (CSCN; ~5-10 PI’s) used different technologies to develop experiments: Psychopy, Qualtrics, Limesurvey, jsPsych, etc.

Each of these has advantages and disadvantages.

Mostly, pragmatic aspects guided the decision: lab history and resources, coding experience, type of experiment (EEG/behavioral, lab/online), …

Issues

Each protocol started almost from scratch. Sometimes a single task would define the technology used.

At some point, we had multiple implementations of the same tasks in different technologies, not always exact replicas.

Some would work in certain computers, other did not. Output data wildly different.

Issues Survey



  • Experiments
  • Resources
  • Reproducibility

2 questions voluntary survey:

Experiment issues

  • Errors in experiment logic
  • Errors in items coding
  • Data not what we expected
  • Data structure makes data preparation hard
  • Match between hypotheses and data not clear
  • Variables or questions not used in the analysis/paper

Resources issues: projects as islands

  • Hours wasted re-programming tasks
  • Thousands of € ‘invested’ in licenses (e.g. Qualtrics)
  • Piloting protocols as a part-time job for Research Assistants
  • Hours wasted re-doing data preparation (each software has its own output format)

Reproducibility issues

  • Anyone knows why this 2012 paradigm/data analysis is not running?
  • Location and organization of projects
  • Data preparation/analyses so ugly, sharing them is hard (let me clean up this a bit before sharing it with you)
  • Idiosyncratic analyses, some of which require licensed closed software (SPSS, Matlab,…)

Our wish list

Our wish list

  • Open source software based on standard technologies
  • Reusable tasks (my project feeds future projects)
  • Based on a mature project or technologies
  • As many ‘automagic’ things as possible
  • Easy to create and analyze paradigms
  • Balancing participants
  • Online/offline

The present

jsPsychR

Open source tools to help create experimental paradigms with jsPsych, simulate participants and standardize the data preparation and analysis

1) jsPsychMaker

2) jsPsychMonkeys

3) jsPsychHelpeR

Goal

A big catalog of tasks in jsPsychMaker. Each task runs with jsPsychMonkeys to create virtual participants, and have a script in jsPsychHelpeR to automate data preparation (re-coding, reversing items, calculating dimensions, etc.).

The final goal is to help you have the data preparation and analysis ready before collecting any real data, reducing errors in the protocols, and making the move towards registered reports easier.

So far

  • 3 main R packages (jsPsychMakeR, jsPsychMonkeys, jsPsychHelpeR)
  • 1 R package for Administration tasks (jsPsychAdmin)
  • > 80 pages manual
  • ~100 tasks ready (with maker and helper scripts, plus original paper for most)
  • > 30 online protocols with > 5000 participants (Prolific Academic, Social Media, etc.)
  • A number of offline (lab) protocols and participants
  • Used by researchers in Chile, Colombia, Spain
  • Everything is open source: https://github.com/gorkang/jsPsychRmanual
  • 2 publications using the system (50% Registered reports) + more in the pipeline…
  • So many errors caught early… ( ͡ᵔ ͜ʖ ͡ᵔ )

The team

Current developers:

  • Gorka Navarrete, Herman Valencia

Initial idea and development:

  • Gorka Navarrete, Nicolas Sanchez-Fuenzalida, Nicolas Alarcón, Alejandro Cofre, Herman Valencia

Discussions, ideas, testing:

  • Esteban Hurtado, Alvaro Rivera, Juan Pablo Morales, …

jsPsychR tools



1) jsPsychMaker

2) jsPsychMonkeys

3) jsPsychHelpeR

jsPsychMaker

Features jsPsychMaker

  • Fully open source, based on web standards (jsPsych)
  • Reuse ~ 100 tasks
  • Online and offline protocols
  • Balancing of participants to between participants conditions
  • Easy to create new tasks
  • Full control over order or tasks (randomization, etc.)
  • Participants can continue where they left (or not)
  • Time and number of participants limits
  • Multilingual support (for a selected number of tasks)
  • All the parameters can be quickly changed editing a single file

Available tasks

Create New Tasks

Creating new tasks is as simple as:

  1. Copy example tasks to your computer
jsPsychMaker::copy_example_tasks(
  destination_folder = "~/Downloads/ExampleTasks"
  )
  1. Copy paste items, adapt csv/excel file

  2. Create protocol (see next slide)

Create protocol

Create a protocol with three existing tasks plus the adapted Example tasks:

jsPsychMaker::create_protocol(
  canonical_tasks = c("AIM", "EAR", "IRI"),
  folder_tasks = "~/Downloads/ExampleTasks/",
  folder_output = "~/Downloads/protocol999",
  launch_browser = FALSE
)

jsPsychR tools



1) jsPsychMaker

2) jsPsychMonkeys

3) jsPsychHelpeR

jsPsychMonkeys

 

Features jsPsychMonkeys

  • Fully open source (R, docker, selenium)
  • Online and offline
  • Sequentially and in parallel
  • Get pictures of each screen
  • Store logs to make debugging easier
  • Watch the monkeys as they work for you
  • Random pauses or refreshing to simulate human behavior
  • Set random seed to make the monkey’s behavior consistent

Release monkeys

Release a single Monkey and take a look:

jsPsychMonkeys::release_the_monkeys(
  uid = 11,
  initial_wait = 0,
  wait_retry = 0,
  local_folder_tasks = "~/Downloads/protocol999/",
  open_VNC = TRUE
)

Release 10 Monkeys in parallel:

jsPsychMonkeys::release_the_monkeys(
  uid = 1:10,
  sequential_parallel = "parallel",
  number_of_cores = 10,
  local_folder_tasks = "~/Downloads/protocol999/",
  open_VNC = FALSE
)

jsPsychR tools



1) jsPsychMaker

2) jsPsychMonkeys

3) jsPsychHelpeR

jsPsychHelpeR

Features jsPsychHelpeR

  • Fully open source (R)
  • Get tidy output data frames for each task, and for the whole protocol
  • Standard naming for tasks, dimensions, scales, …
  • Include tests for common issues
  • Snapshots to detect changes in data processing
  • Functions to help create new tasks correction using a standard template
  • Automatic reports with progress, descriptive statistics, code-book, etc.
  • Create a fully reproducible Docker container with the project’s data preparation and analysis
  • Create a blinded data frame to perform blinded analyses

jsPsychHelpeR

Create project for data preparation:

jsPsychHelpeR::run_initial_setup(
  pid = 999,
  data_location = "~/Downloads/protocol999/.data/",
  folder = "~/Downloads/jsPsychR999"
  )

Create a task correction script:

jsPsychHelpeR::create_new_task("MultiChoice")

Visualize and run data preparation:

targets::tar_visnetwork(targets_only = TRUE, label = "time")

targets::tar_make()

Challenge: everything in 3 minutes?

Create protocol, simulate participants and prepare data…

# Full process
rstudioapi::navigateToFile("R/script-full-process.R")


Survey Experiment Issues

Survey results

Let’s try to download the data, process it and show a report with the results:


Plan A: run Experiment Issues project

rstudioapi::openProject("../jsPsychHelpeR-ExperimentIssues/jsPsychHelpeR-ExperimentIssues.Rproj", newSession = TRUE)



Plan B: If something fails, we always have the monkeys!

browseURL("../jsPsychHelpeR-ExperimentIssues/outputs/reports/report_analysis_monkeys.html")

Limitations

  • Very easy to create new scales, and simple tasks, but complex experimental tasks require javascript and HTML knowledge (although there are a good number of examples available)

  • Data preparation for new experimental tasks requires expertise in R (simple surveys not so much)

  • Analysis reports require some R knowledge (simple templates available)

  • Requires access to a server for online tasks

  • Only behavioral tasks

The future

Too many things, too little time

  • Create templates for most common experimental designs
  • Templates for data preparation and analysis of common experimental designs
  • More tasks, translations, tests, …
  • So far, development based in our needs (and possibilities)
  • Upgrade to latest jsPsych when available
  • Improve, clean, share…
  • Publish jsPsychR paper

Help

  • Javascript programmers

  • R programmers

  • Task creators

  • Testers

Back to Registered reports




Can jsPsych really help? (1/2)

  • With jsPsychR, protocols are standardized and with (mostly) clean code

    • Less errors in protocols
  • Data preparation is 90% automatic, standardized, and beautiful

    • Less errors in data preparation
  • When errors are found and fixed, old protocols can benefit from the corrections, old results can be checked, …

Can jsPsych really help? (2/2)

  • Super easy to work on analysis before collecting human data

  • Much easier to write up a good analysis plan, share it, improve it, …

  • Sharing protocol, materials, data preparation is trivial (single command)

  • Creating future proof full reproducible projects (with Docker) is one command away

RR + jsPsychR

More information

For more information about RR, templates, checklists, list of participating journals (>300), etc.:

Also, check out the future:

And our manual:

References

Baker, Monya. 2016. “Reproducibility Crisis.” Nature 533 (26): 353–66.
Nosek, Brian A, Tom E Hardwicke, Hannah Moshontz, Aurélien Allard, Katherine S Corker, Anna Dreber, Fiona Fidler, et al. 2022. “Replicability, Robustness, and Reproducibility in Psychological Science.” Annual Review of Psychology 73: 719–48.
Open-Science-Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251): aac4716.
Rubin, Mark. 2017. “An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach.” Review of General Psychology 21 (4): 321–29.
Soderberg, Courtney K, Timothy M Errington, Sarah R Schiavone, Julia Bottesini, Felix Singleton Thorn, Simine Vazire, Kevin M Esterling, and Brian A Nosek. 2021. “Initial Evidence of Research Quality of Registered Reports Compared with the Standard Publishing Model.” Nature Human Behaviour 5 (8): 990–97.

Thanks!



Gorka Navarrete

gorkang@gmail.com

https://fosstodon.org/@gorkang